I know that RF stands for radio frequency but what chip in the Atari handles the video output and does anybody know if something similar would work on a modern tv?
Devices of this era were designed to interface with common TV sets of the day. These used (mostly) one of two systems, NTSC (North America) and PAL (Europe and much of the rest of the world). These systems were for the VIDEO format but TV sets of the time rarely had direct video inputs. The only input on most TVs was "antenna" or RF input from an antenna or cable.
Often there were two screw terminals for a 300 Ohm twin-lead connection that was common. Later TVs added a coax connector that was used with cable TV converter boxes and later VCRs.
So the problem was how to get the video generated by a game console onto the screen of a TV. The answer was what is called an RF MODULATOR. These could be internal to the unit in which case there was either a coax output or a twin-lead output that could be connected to the TV. Other manufacturers chose to leave the RF modulator out of the unit and supplied or required an external RF modulator device. This took an input from the game console and converted it to RF usually on TV channel 3 or 4 whichever was not used by a local TV station.
As far as Atari goes, it's not clear which exact console or device you are talking about and they produced many different ones over the years and as I recall they used both internal and external RF modulators as well as later models that offered higher quality RGB outputs to get around the limitations of NTSC/PAL for those using their devices with dedicated computer monitors which had become common a few years after these devices were introduced.
Today, however, RF modulators have all but disappeared. The main reason is that HDMI is ubiquitous on TV sets of all sizes and that is very easy to add to any video-producing device.
Old home computers made use of an off-the shelf RF modulator component.
These were ready-made analog boards (one prominent manufacturer was Astec, as in the picture) that converted Composite Video into an RF signal suitable for the home-TV's RCA input.
Because most modern TVs no longer have analogue input, such an approach wouldn't work today.