Many TV designs up into the 1970s were so called live chassis designs, which used one leg of the mains input as a reference ground. This saved materials and weight - given some early color TVs used 200+ watts at 100% duty cycle, you would have needed a rather bulky and heavy transformer, given that PSMPS technology was not really mature for consumer devices at that time. Some sets used a small transformer to supply some low voltage circuitry, while going straight off the mains for other parts of the unit - but still having the common ground, even of the transformer supplied parts, directly connected to the mains.
An RGB input is a DC coupled input, unlike an RF input.
Most home electrical systems do not use polarized plugs, or the correct polarization of plugs and sockets cannot be relied upon sufficiently to use it as a safety feature.
A DC coupled input needs a DC coupled ground - which, in a live chassis design, has a 50% chance (with an unpolarized plug) to be at 120V/240V mains live potential....
Thus, a live chassis design CANNOT have any DC coupled inputs or outputs to random external devices*, unless complex isolation circuitry (which is not trivial for a wideband and DC coupled signal like RGB video) is used.
(There is hearsay that a significant amount of people got injured attempting to retrofit audio outputs, RGB or composite inputs etc. to live chassis TVs back in the day.)
*Actually, there were some live chassis RADIOS too, which sometimes had special connectors for turntables or microphones that were in themselves completely insulated.... This kind of design would be considered insane today.
And there is yet another reason. Some color TV designs did not, anywhere in the circuitry, decode the received signal into RGB either, instead taking advantage of multiple control inputs on a CRT (eg cathode input for low bandwith chroma difference signals, grids for high bandwidth luminance) to only compose the "final" picture output within the CRT itself.
Addendum: One might wonder how so much electronics was powered "straight off the mains". Keep in mind that TVs until the late 70s often used at least some vacuum tube circuitry - and vacuum tube circuitry works great off a +150...250VDC bus, which you can relatively easily create from mains input without the use of a transformer. And even in a fully semiconductor based design, quite some of the power hungry circuitry is used to drive the CRT and the EHT inverter (usually combined into one circuit with the horizontal output stage) - this is typically not low voltage circuitry either (which is the reason why vacuum tube circuits were used for this for a long time - high voltage transistors or thyristors were LESS economical to use then. Fascinatingly, there were production color TV designs that use ICs and vacuum tubes together in one chassis.)