TV manufacturers didn't have a single, obvious RGB connection standard to implement. Physically, there was SCART (with competing European and Japanese pinouts), RCA, DE-9, and various manufacturer-specific DIN plugs to choose from.
Then you have the various electrical signals to send over them such as RGBS, RGsB, RGBHV, YPrPb, digital RGBI, etc.
And VCRs didn't even need RGB because Y/C was good enough (in fact, I think Y/C is good enough for early computers, also), and if you're recording off the air, the quality wasn't that great to begin with. (Prerecorded videocassettes didn't become affordable until later.)
While this was all being sorted out, device manufacturers typically provided an RF Modulator that could work with any TV through the antenna connector that every television already equipped (for example, 300 ohm twin lead screw connectors or a coax connector of some sort). So for the TV manufacturers, their job was already done. Providing more connectors for external devices quickly hits diminishing returns.